Computer systems use memory devices, such as dynamic random access memory (“SDRAM”) devices, to store instructions and data that are accessed by a processor. In a typical computer system, the processor communicates with the system memory through a processor bus and a memory controller. The processor issues a command, such as a read command, and an address designating the location from which data or instructions are to be read. The memory controller uses the command and address to generate appropriate command signals as well as row and column addresses, which are applied to the system memory. In response to the commands and addresses, data are transferred between the system memory and the processor. The memory controller is often part of a system controller, which also includes bus bridge circuitry for coupling the processor bus to an expansion bus, such as a PCI bus.
Although the operating speed of memory devices has continuously increased, this increase in operating speed has not kept pace with increases in the operating speed of processors. Even slower has been the increase in operating speed of memory controllers coupling processors to memory devices. The relatively low speed of memory controllers and memory devices limits the communication bandwidth between the processor and the memory devices.
One approach to increasing memory bandwidth is to use multiple memory modules coupled to the processor through a memory controller or the like. Each of the memory modules can have a memory hub architecture in which a memory hub is coupled to several memory devices, such as DRAM devices. The memory hub in each of the memory modules can be coupled to the memory controller through a plurality of high-speed bit-lanes. Computer systems employing this architecture can have a higher bandwidth because a processor can access one memory device while another memory device is responding to a prior memory access. For example, the processor can output write data to one of the memory devices in the system while another memory device in the system is preparing to provide read data to the processor.
A conventional computer system 100 having a memory hub architecture is shown in FIG. 1. The computer system 100 includes a processor 104 for performing various computing functions, such as executing specific software to perform specific calculations or tasks. The processor 104 includes a processor bus 106 that normally includes an address bus, a control bus, and a data bus. The processor bus 106 is typically coupled to cache memory 108, which, as previously mentioned, is usually static random access memory (“SRAM”). Finally, the processor bus 106 is coupled to a system controller 110, which is also sometimes referred to as a “North Bridge.”
The system controller 110 serves as a communications path to the processor 104 for a variety of other components. More specifically, the system controller 110 includes a graphics port that is typically coupled to a graphics controller 112, which is, in turn, coupled to a video terminal 114. The system controller 110 is also coupled to one or more input devices 118, such as a keyboard or a mouse, to allow an operator to interface with the computer system 100. Typically, the computer system 100 also includes one or more output devices 120, such as a printer, coupled to the processor 104 through the system controller 110. One or more data storage devices 124 are also typically coupled to the processor 104 through the system controller 110 to allow the processor 104 to store data or retrieve data from internal or external storage media (not shown). Examples of typical storage devices 124 include hard and floppy disks, tape cassettes, and compact disk read-only memories (CD-ROMs).
The system controller 110 also includes a memory hub controller 128 for controlling access to several system memory modules 130a-n. Each of the memory modules 130a-d includes a substrate 134 on which a memory hub 140 and a plurality of memory devices 148 are mounted. The memory devices 148 may be SDRAM devices or some other type of memory devices. The memory devices 148 are coupled to the memory hub 140 through a bus system 150 that normally includes a command bus, an address bus and a data bus. The memory hub controller 128 is coupled to the memory hubs 140 in the respective memory modules 130a-d through several high speed downstream bit-lanes 162 and several high speed upstream bit-lanes 164. Each of the downstream bit-lanes 162 includes a signal line to couple signals from the memory hub controller 128 to the memory hubs 140 in the memory modules 130a-n. Similarly, each of the upstream bit-lanes 164 includes a signal line to couple signals from the memory hubs 140 in the memory modules 130a-n to the memory hub controller 128. In the computer system 100 example of FIG. 1, the signals coupled through the bit lanes 162, 164 are in the form of packets. The packets coupled through the downstream bit-lanes 162 generally include memory commands, addresses and write data. The packets coupled through the upstream bit-lanes 164 generally include read data and acknowledgement signals. However, if the memory hubs 140 include the capability of reading data from or writing data to other memory modules 130a-n, the packets coupled through either of the bit-lanes 162, 164 may include memory commands, addresses, write data, read data and acknowledgment signals. However, other means of coupling signals between the memory hub controller 128 and the memory modules 130a-n, including dedicated command, address, write data and read data lines. In the computer system 100 shown in FIG. 1, there are 10 downstream bit lanes 162a-j and 14 upstream bit lanes 164a-n. However, different numbers of bit lanes 162, 164 may be used. However, in several of the examples described below, only 4 bit lanes 162a-d are shown and described for purposes of clarity and simplicity.
The bit-lanes 162, 164 are divided into segments between respective memory modules 130a-n. More specifically, the memory hub controller 128 is coupled to the memory hub 140 in the first memory module 130a through a first segment 170 of the bit-lanes 162, 164 and through a connector 190 on the substrate 134. The memory hub 140 in the second memory module 130b is connected to the memory hub controller 128 through a second set of bit-lane terminals of the memory hub 140 in the first memory module 130a, which are connected to a second connector 192 on the substrate 134. The signals are thus coupled between the memory hub 140 in the second memory module 130b using the first segment 170 of the bit-lanes 190, 192 as well as a second segment 172 of the bit-lanes 190, 192. Similarly, the memory hubs 140 in the remaining N−2 memory modules 130 are connected to the memory hub controller 128 in the same manner through the first and second memory modules 130a,b, the first and second bit-line segments 170, 172, respectively, and through any intervening memory modules 130 and bit-line segments.
In operation, the memory hub 140 in each of the memory modules 130a-c can processes the signals coupled through the bit-lanes 160-166 in one of two ways. First, if signals emanating “downstream” from the memory module 130 are being coupled to the memory hub controller 128 or signals from the memory hub controller 128 are being coupled to a downstream memory module 130, the memory hub 140 in the memory module simply passes the signals from each of the bit-lanes 162, 164 in one segment to the corresponding bit-lanes 162, 164 in the other segment. However, if the signals coupled through the bit-lanes 162, 164 are for accessing one or more of the memory devices 148 on the memory module 130, the signals are coupled to and from the memory hub 140 for that module 130. The memory hub 140 then uses signals received from the memory hub controller 128 to generate signals for accessing the memory devices 148.
The memory hub architecture used in the computer system 100 allows the processor 104 to more efficiently write data to and read data from each of the memory devices 148. For example, the processor 104 can issue a read command to an address in a memory device 148 in the memory module 130a. During the time that the memory hub 140 in the memory module 130a issues a corresponding read command to the addressed memory device 148 in the memory module 130a and the memory device 148 responds to the read command, the processor 104 can issue a memory command to a different memory module 130, or it can perform a function other than accessing a memory device. When the read data is ready to be sent from the memory module 130a, the processor can again communicate with the memory hub 140 in the memory module.
Processor-based systems using a memory hub architecture can have configurations other than the configuration shown in FIG. 1. For example, devices other than the processor 104 may access the memory modules 130 through the memory hub controller 128. Also, the memory hub controller 128 may be a stand-alone unit or physically included as a part of components of an electronic system other than the system controller 110. Other modifications and variations will be apparent to one skilled in the art.
Each of the downstream bit-lanes 162 or each of the upstream bit-lanes 164 may be each assigned to pass specific signals, such specific bits of a packet. However, the bit-lanes 162, 164 are also sometimes assigned on a flexible basis so that, if all or a portion of a bit-lane becomes defective, another bit-lane can be substituted for the defective bit-lane. In fact, in some systems, an extra bit-lane can be provided specifically for this purpose. One example of the flexible assignment of bit-lanes is schematically illustrated in the portion of the computer system 100 shown in FIG. 2. As shown in FIG. 2, the bit-lane 162c in the first segment 170 has become defective for one of a variety of reasons. The conductor for the bit-lane 162c may have become open-circuited or short-circuited, or an electrical component connected to the bit-lane 162c in the memory hub controller 128 or in the memory hub 140 in the memory module 130a may have become defective. In such case, the memory hub controller 128 re-routes signals that would normally pass through the bit-lane 162c to the extra bit-lane 162d. The computer system 100 can therefore continue to function despite the inoperable bit-lane 162c in the first bit-line segment 170.
The substitution of an extra bit-lane 162d for a defective bit-lane 162c works well if only one bit-lane is inoperable. However, if two or more bit-lanes have become inoperable, this substitution procedure will not suffice unless two extra bit-lanes have been provided. For example, as shown in FIG. 3, the bit-lane 162c has become inoperable in the first bit-line segment 170, and the bit-lane 162b has become inoperable in the second bit-line segment 172. In such case, only two bit-lanes 162a, 162d will remain operable to pass signals from the memory modules 130b-c, although three bit-lanes remain operable to pass signals from the memory module 130a. However, the system shown in FIG. 3 generally becomes inoperable if two or more bit-lanes 162 in any segment 170-176 becomes inoperable.
There is therefore a need for a memory system architecture that allows a memory system to be repaired by bit-lane substitution despite failures in multiple bit-lanes.